Certain computing architectures (such as the x86 architecture) do not require that program instructions be of a certain length or begin or end on predefined boundaries. Such architectures permit execution of random byte sequences within programs, even if the execution of such byte sequences violates an instruction alignment generated by a compiler, so long as the byte sequences that follow represent viable instructions. For example, a program's stack may point to and execute the second byte in a five-byte instruction, so long as the bytes that follow the second byte represent viable instructions.
Because of this, researchers have determined that malicious programmers may exploit legitimate programs by identifying and executing out-of-alignment byte sequences within a program (i.e., byte sequences that, when executed, violate an instruction alignment generated by a compiler) that may result in viable instructions. Researchers have determined that malicious programmers may identify such out-of-alignment byte sequences by: 1) analyzing the address space associated with a legitimate process, 2) identifying a byte or series of bytes within the address space that may result in a control-transfer instruction (such as a return instruction) that may direct control flow of the process, and then 3) determining whether any of the bytes that precede the control-transfer instruction represent viable and potentially useful instructions (such as system calls).
Malicious programmers may then create shell code that directs control flow of the program to jump to each useful byte sequence. For example, a malicious programmer may create shell code that exploits a buffer overflow in a legitimate program in order to overwrite the program's stack. The shell code may then cause the overwritten stack to return a value that jumps to an out-of-alignment byte sequence within the legitimate program that executes an instruction desired by the malicious programmer (such as a system call to create a file, to open a file, or to write to a memory location). The shell code may then cause the overwritten stack to continue to make returns to byte sequences that perform the various functions desired by the malicious programmer. In this way, the malicious programmer may, without inserting any code, cause a legitimate program to perform an unintended, and potentially malicious, action.